Systems and methods for constraining a virtual reality surgical system

ABSTRACT

A method of operating a surgical control system comprises displaying an image of a surgical environment, from a field of view of an imaging instrument, on a display system. The display system is configured for mounting to a head of a user. The method also includes detecting an imaging control input from the user and responsive to the detection of the imaging control input, enabling an imaging control mode of the surgical control system. The method also includes detecting a movement of the head of the user. While the surgical control system is in the imaging control mode and responsive to the user&#39;s head movement, an image of the surgical environment from a changed field of view of the imaging instrument is displayed. The changed field of view corresponds to the detected movement of the user&#39;s head.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is the U.S. national phase of InternationalApplication No. PCT/US2018/028376, filed Apr. 19, 2018, which designatedthe U.S. and claims priority to and benefit of the filing date of U.S.Provisional Patent Application 62/487,833, filed Apr. 20, 2017, whichare incorporated by reference herein in its their entirety.

FIELD

The present disclosure is directed to systems and methods for performinga medical procedure or training for a medical procedure using a virtualreality display system and more particularly to systems and methods forproviding constraints within a surgical system or surgical trainingsystem when using a virtual reality display system.

BACKGROUND

Minimally invasive medical techniques are intended to reduce the amountof tissue that is damaged during invasive medical procedures, therebyreducing patient recovery time, discomfort, and harmful side effects.Such minimally invasive techniques may be performed through naturalorifices in a patient anatomy or through one or more surgical incisions.Through these natural orifices or incisions, clinicians may insertmedical tools to reach a target tissue location. Minimally invasivemedical tools include instruments such as therapeutic instruments,diagnostic instruments, and surgical instruments. Minimally invasivemedical tools may also include imaging instruments such as endoscopicinstruments that provide a user with a field of view within the patientanatomy. Some minimally invasive medical tools and imaging instrumentsmay be teleoperated or otherwise computer-assisted. When performingteleoperational procedures or providing training simulations forteleoperational procedures, head-mounted display systems may be used toprovide the clinician with an immersive augmented-reality or virtualreality experience. System controls are needed to provide for safe andreliable operation of teleoperational systems when using an immersivedisplay system.

SUMMARY

The embodiments of the invention are summarized by the claims thatfollow below.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory innature and are intended to provide an understanding of the presentdisclosure without limiting the scope of the present disclosure. In thatregard, additional aspects, features, and advantages of the presentdisclosure will be apparent to one skilled in the art from the followingdetailed description.

In one embodiment a method of operating a surgical control systemcomprises displaying an image of a surgical environment, from a field ofview of an imaging instrument, on a display system. The display systemis configured for mounting to a head of a user. The method also includesdetecting an imaging control input from the user and responsive to thedetection of the imaging control input, enabling an imaging control modeof the surgical control system. The method also includes detecting amovement of the head of the user. While the surgical control system isin the imaging control mode and responsive to the user's head movement,an image of the surgical environment from a changed field of view of theimaging instrument is displayed. The changed field of view correspondsto the detected movement of the user's head.

In another embodiment, a method of operating a surgical control systemcomprises generating an image of a surgical environment from a viewpointof an imaging tool having a field of view. A surgical instrument ispositioned within the surgical environment. The method also includesdisplaying the image of a surgical environment on a display system. Thedisplay system is configured for mounting to a head of a user. Themethod also includes detecting a movement of the head of the user anddetermining if the movement of the user's head is within a boundarycorresponding to the imaging tool field of view. If the movement of theuser's head is within the boundary, the image of the surgicalenvironment on the display system is changed by generating a changedviewpoint in the imaging tool field of view that corresponds with thedetected movement of the user's head.

In another embodiment, a method of operating a surgical control system,the method comprises generating an image of a surgical environment. Asurgical instrument is positioned within the surgical environment and iscoupled to a manipulator arm having a range of motion. The method alsoincludes displaying the image of a surgical environment on a displaysystem. The display system is configured for mounting to a head of auser. The method also includes detecting a manipulator arm input motionfrom the user and determining if the manipulator arm input motioncorresponds to movement of the manipulator arm within the range ofmotion. If the movement of the manipulator arm input motion correspondsto movement outside the range of motion of the manipulator arm, a ghostimage of the surgical instrument is generated, wherein the ghost imagemoves in correspondence with the manipulator arm input motion.

BRIEF DESCRIPTIONS OF THE DRAWINGS

Aspects of the present disclosure are best understood from the followingdetailed description when read with the accompanying figures. It isemphasized that, in accordance with the standard practice in theindustry, various features are not drawn to scale. In fact, thedimensions of the various features may be arbitrarily increased orreduced for clarity of discussion. In addition, the present disclosuremay repeat reference numerals and/or letters in the various examples.This repetition is for the purpose of simplicity and clarity and doesnot in itself dictate a relationship between the various embodimentsand/or configurations discussed.

FIG. 1 a provides a view of a surgical environment including ateleoperational surgical system including an immersive display system.

FIG. 1 b provides a view of a simulated surgical environment including ateleoperational surgical system including an immersive display system.

FIG. 2 provides a field of view image from an imaging instrument.

FIG. 3 illustrates a hand controller of a teleoperational surgicalsystem.

FIG. 4 is a flowchart providing a method for entering and exiting avirtual explore mode of the teleoperational surgical system.

FIGS. 5-8 illustrate a surgical environment, an external environment,and displayed images on an immersive display system.

FIG. 9 is a flowchart providing a method for operating within an imagingcontrol mode of the teleoperational surgical system.

FIG. 10 illustrates a surgical environment, an external environment, anddisplayed images on an immersive display system.

FIG. 11 illustrates a method for entering and operating within a virtualmanipulator arm mode of a teleoperational system.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of thepresent disclosure, reference will now be made to the embodimentsillustrated in the drawings, and specific language will be used todescribe the same. It will nevertheless be understood that no limitationof the scope of the disclosure is intended. In the following detaileddescription of the aspects of the invention, numerous specific detailsare set forth in order to provide a thorough understanding of thedisclosed embodiments. However, it will be obvious to one skilled in theart that the embodiments of this disclosure may be practiced withoutthese specific details. In other instances well known methods,procedures, components, and circuits have not been described in detailso as not to unnecessarily obscure aspects of the embodiments of theinvention.

Any alterations and further modifications to the described devices,instruments, methods, and any further application of the principles ofthe present disclosure are fully contemplated as would normally occur toone skilled in the art to which the disclosure relates. In particular,it is fully contemplated that the features, components, and/or stepsdescribed with respect to one embodiment may be combined with thefeatures, components, and/or steps described with respect to otherembodiments of the present disclosure. In addition, dimensions providedherein are for specific examples and it is contemplated that differentsizes, dimensions, and/or ratios may be utilized to implement theconcepts of the present disclosure. To avoid needless descriptiverepetition, one or more components or actions described in accordancewith one illustrative embodiment can be used or omitted as applicablefrom other illustrative embodiments. For the sake of brevity, thenumerous iterations of these combinations will not be describedseparately. For simplicity, in some instances the same reference numbersare used throughout the drawings to refer to the same or like parts.

The embodiments below will describe various instruments and portions ofinstruments in terms of their state in three-dimensional space. As usedherein, the term “position” refers to the location of an object or aportion of an object in a three-dimensional space (e.g., three degreesof translational freedom along Cartesian X, Y, Z coordinates). As usedherein, the term “orientation” refers to the rotational placement of anobject or a portion of an object (three degrees of rotationalfreedom—e.g., roll, pitch, and yaw). As used herein, the term “pose”refers to the position of an object or a portion of an object in atleast one degree of translational freedom and to the orientation of thatobject or portion of the object in at least one degree of rotationalfreedom (up to six total degrees of freedom).

Referring to FIG. 1 a of the drawings, a surgical environment 10includes a teleoperational medical system 11 for use in, for example,medical procedures including diagnostic, therapeutic, or surgicalprocedures. The teleoperational medical system generally includes ateleoperational assembly mounted to or near an operating table O onwhich a patient P is positioned. The teleoperational assembly mayinclude one or more modular manipulator arms 12. A medical instrumentsystem 14 or an imaging system 15 may be operably coupled to ateleoperational manipulator (e.g. an arm) of the teleoperationalassembly. The imaging system may be, for example, a stereoscopicendoscope imaging system. An operator input system 16 allows a surgeon Sor other type of clinician to control the operation of the medicalinstrument system 14 and/or the imaging system 15. One or more assistantsurgeons, anesthesiologists, or support personnel may also be present inthe surgical environment.

The teleoperational medical system also includes a display system 17which may present images captured by the imaging system 15, surgicalnavigation and guidance images, and/or alphanumeric or symbolicinformation to assist the surgeon or assistants with the surgicalprocedure. The display system may be, for example, an immersive displaysystem worn by the surgeon S. More specifically, the immersive displaysystem may be a head-mounted display system for presenting an image ofthe interior anatomical environment from the imaging system 15,augmented or virtual images of the interior anatomical environment,images from external of the patient anatomy, interactive userinterfaces, or other image-based, graphical, or textual information tothe surgeon S. The head-mounted display system may be worn as a set ofglasses or goggles covering both of a user's eyes. FIG. 2 illustrates afield of view image 30 from an imaging instrument 15 positioned withinthe interior anatomy of the patient P. The image 30 may be displayed onthe display system 17. With stereoscopic image data, multiple viewpointsmay be generated from the field of view stereoscopic image data whilethe imaging instrument is held in a static position. Thus, with animmersive display system, the field of view may be presented withslightly different viewpoint images as the user's head moves the displaysystem. This may provide the user with a more immersive experience whenviewing the field of view.

The teleoperational medical system also includes a control system 20 incommunication with the operator input system 16, the teleoperationalassembly and the display system 17, as described below.

In this embodiment, the operator input system 16 includes one or a setof operator hand controllers 18 (FIG. 3 ) for controlling one ormultiple medical instrument systems 14 or the endoscopic imaging system15. The input system also may include other types of user inputsincluding pedal inputs, gaze tracking, voice command recognition, headgesture recognition. In various alternatives the operator handcontrollers 18 may be tethered by power and/or signal transmissioncabling or may be untethered/wireless. As shown in FIG. 3 , the operatorcontroller 18 may include one or more of any number of a variety ofinput devices such as grip inputs 22 and trigger switch 24. The inputdevices may be used to, for example, move a distal end of an endoscopicimaging system within the patient P, close grasping jaw end effectors,apply an electrical potential to an electrode, deliver a medicinaltreatment, or the like. In various alternatives, the operator inputsystem may additionally or alternatively include joysticks, trackballs,data gloves, trigger-guns, hand or foot-operated controllers, voicerecognition and control devices, touch screens, body motion or presencesensors, and the like. In some embodiments, the hand controller 18 willbe provided with the same degrees of freedom as the medical instrumentsof the teleoperational assembly to provide the surgeon withtelepresence, the perception that the control device(s) are integralwith the instruments so that the surgeon has a strong sense of directlycontrolling instruments as if present at the surgical site. In otherembodiments, the hand controller 18 may have more or fewer degrees offreedom than the associated medical instruments and still provide thesurgeon with telepresence.

The manipulator arms 12 support and manipulate the medical instrumentsystem 14 and/or the imaging system 15 while the surgeon S conducts theprocedure from the patient side or another location within the surgicalenvironment. The number of medical instrument systems 14 used at onetime will generally depend on the diagnostic or surgical procedure andthe space constraints within the operating room among other factors.Each arm 12 of the teleoperational assembly may include a kinematicstructure of one or more servo or non-servo controlled links. Theteleoperational arm 12 may also include a plurality of motors that driveinputs on the medical instrument system 14 or imaging system 15. Thesemotors move in response to commands from the control system 20. Themotors include drive systems which when coupled to the medicalinstrument system 14 or imaging system 15 may move the systems into orout of a naturally or surgically created anatomical orifice. Othermotorized drive systems may move the distal end of the systems inmultiple degrees of freedom, which may include three degrees of linearmotion (e.g., linear motion along the X, Y, Z Cartesian axes) and inthree degrees of rotational motion (e.g., rotation about the X, Y, ZCartesian axes). Additionally, the motors can be used to actuate anarticulable end effector of the instrument for grasping tissue in thejaws of a biopsy device or the like. Instruments 14 may include endeffectors having a single working member such as a scalpel, a bluntblade, an optical fiber, or an electrode. Other end effectors mayinclude, for example, forceps, graspers, scissors, or clip appliers.

The control system 20 includes at least one memory and at least oneprocessor, and typically a plurality of processors, for effectingcontrol between the medical instrument system, the imaging system 15,the operator input system 16, the display system 17, and other auxiliarysystems which may include, for example, additional imaging systems,audio systems, fluid delivery systems, display systems, illuminationsystems, steering control systems, irrigation systems, and/or suctionsystems. The control system 28 also includes programmed instructions(e.g., a computer-readable medium storing the instructions) to implementsome or all of the methods described in accordance with aspectsdisclosed herein. While control system 20 is shown as a single block inthe simplified schematic of FIG. 1 a , the system may include two ormore data processing circuits with one portion of the processingoptionally being performed on or adjacent the teleoperational assembly,another portion of the processing being performed at the operator inputsystem 16, and the like. The control system 20 may employ any of a widevariety of centralized or distributed data processing architectures.Similarly, the programmed instructions may be implemented as a number ofseparate programs or subroutines, or they may be integrated into anumber of other aspects of the teleoperational systems described herein.In one embodiment, control system 20 supports wireless communicationprotocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, andWireless Telemetry.

In some embodiments, control system 20 may include one or more servocontrollers that receive force and/or torque feedback from the medicalinstrument system or the imaging system. Responsive to the feedback, theservo controllers transmit signals to the operator input system 16. Theservo controller(s) may also transmit signals instructingteleoperational assembly to move the medical instrument system(s) 14and/or endoscopic imaging system 15 which extend into an internalsurgical site within the patient body via openings in the body. Anysuitable conventional or specialized servo controller may be used. Aservo controller may be separate from, or integrated with,teleoperational assembly. In some embodiments, the servo controller andteleoperational assembly are provided as part of a teleoperational armpositioned adjacent to the patient's body.

The control system 20 can be communicatively coupled with the imagingsystem 15 and can include a processor to process captured images forsubsequent display, such as to a surgeon on the surgeon's console, or onanother suitable display located locally and/or remotely. For example,where a stereoscopic endoscope is used, the control system 20 canprocess the captured images to present the surgeon with coordinatedstereo images of the surgical site. Such coordination can includealignment between the opposing images and can include adjusting thestereo working distance of the stereoscopic endoscope.

As shown in FIG. 1 b , the control system 20, operator input system 16and immersive display system 17 may also be used to conduct a procedurewith a simulated surgical environment 40. The simulated surgicalenvironment 40 may be viewed on the immersive display system 17 by theuser S. In one embodiment, the simulated surgical environment is a threedimensional, virtual-reality environment representing a surgicalenvironment that may be used for designing and providing cliniciantraining. The field of view of a simulated imaging system may appearsimilar to a live field of view image 30. The simulated surgicalenvironment may be a pre-recorded image of a patient anatomicalenvironment, a computer-generated illustration of a patient anatomicalenvironment, or a hybrid of the two. In the simulator environment,virtual instruments in the field of view image may be operated using thehand controllers 18. Likewise, the hand controllers may be used toadjust the simulated field of view by moving a simulation imaging system(e.g. a virtual endoscope). In various embodiments, the simulatedsurgical environment may be a complete virtual reality environment inwhich movements of the user S, such as head movements or inputs to theinput system 16, are tied to parameters of virtual components (e.g.,virtual instruments, a virtual imaging system) in the virtual realityenvironment. In these entirely virtual simulated environments, usermovements are not tied to physical components in a physical surgicalenvironment. In some embodiments, user inputs may be provided by trackedgaze or head motion, without the use of hand controllers.

In alternative embodiments, the simulated surgical environment may be aphysical, artificial surgical environment designed to mimic a naturalanatomy or to provide specialized procedure training. In thesealternative embodiments, user inputs may be tied to physical componentsactive in the artificial environment. In still other alternativeembodiments, the simulated surgical environment may be a hybridenvironment including both virtual reality elements and physicalcomponents.

The immersive display systems allow a user to experience a live orsimulated surgical environment as if the user were present within theenvironment. A head mounted immersive display system physically allows auser to have full range of translational and rotational motion that maynot correspond to the physical constraints imposed by the kinematics andsafety features of the teleoperational system. The systems and methodsdescribed below may be implemented to impose constraints on or augmentthe user's experience when using an immersive display system in a liveor simulated surgical environment.

In a traditional teleoperational surgical system in which a user ispositioned at a stationary operator console, the system enters anengagement mode (also known as a “head-in” mode) when sensors detectthat the user's head is positioned in a viewfinder and the user is ableto visualize the surgical environment displayed in the viewfinder viaimages captured by the imaging system. The “following mode” of ateleoperational medical system is a system operational state in whichthe movement of the operator input system, such as the hand controllers,effects movement of the instruments positioned in the patient anatomy.Generally, before the teleoperational medical system can enter thefollowing mode for performing a surgical procedure, the system mustfirst be in the engagement mode. To safely perform a teleoperationalprocedure with a traditional system, movement of the instruments withinthe patient anatomy may be suspended when the operator looks away orotherwise exits the engagement mode, indicating that the display systemis not visible to the operator.

When the display system is not in a fixed position within an operatorconsole, as is the case with an operator-mounted immersive displaysystem, the teleoperational system may be constrained or augmented toprovide a virtual engagement mode that indicates the operator is able tovisualize the surgical environment in the immersive display prior toentering an instrument following mode. FIG. 4 is a flowchart providing amethod 100 for entering and exiting virtual engagement and virtualexplore modes of a teleoperational surgical system. The teleoperationalsurgical system may be, for example a teleoperational surgical system 11for use in a live surgical environment 10 or a simulator environment 40.The method 100 is illustrated in FIG. 4 as a set of operations orprocesses 102 through 114. Not all of the illustrated processes 102through 114 may be performed in all embodiments of method 100.Additionally, one or more processes that are not expressly illustratedin FIG. 4 may be included before, after, in between, or as part of theprocesses 102 through 114. In some embodiments, one or more of theprocesses may be implemented, at least in part, in the form ofexecutable code stored on non-transitory, tangible, machine-readablemedia that when run by one or more processors (e.g., the processors ofcontrol system) may cause the one or more processors to perform one ormore of the processes.

At a process 102, an image of a surgical environment visible in thefield of view of an imaging instrument is displayed. The image isdisplayed on a user-mounted immersive display system. With reference toFIG. 5 , a surgical environment 202 (e.g., an interior anatomy of a livepatient or a simulated surgical environment) may be visible with animaging instrument 204 having a current field of view 206 and apotential field of view 208. The potential field of view may be definedby the range of motion of the imaging instrument 204. An environment 214is external to the surgical environment 202. A user 201 (e.g., a surgeonS) wearing a head-mounted immersive display system 203 (e.g., displaysystem 17) has a view direction 210 toward the current field of view206. A visible image 212 is presented on the display system 203. In thisposition the user may be in a virtual engaged mode. In the virtualengaged mode, the user visualizes the current field of view. From thevirtual engaged mode, the instrument following mode may be entered inwhich instruments in the field of view may be moved under control of theoperator input system.

At a process 104, movement of the user's head is detected. At a process106, a determination is made as to whether the user's head (or a portionof the head such as the eyes) is within a boundary that corresponds toand allows the user to view the field of view. The position and viewdirection of the user's head may be evaluated to determine whether thecurrent field of view is visible to the user.

At a process 108, if the head movement has not exceeded the boundaryand/or the user's head remains directed toward the current field ofview, the image on the display system is still an image of the field ofview 206. If the image data is three dimensional, for example from astereoscopic endoscope instrument, the image of the field of view may bepresented from a slightly different viewpoint corresponding to thedetected movement of the user's head. As shown in FIG. 6 , the head ofthe user 201 has moved, but the user's eye view direction 210 isdirected toward the current field of view 206 and the user's head andeyes are within a boundary 205 corresponding to the field of view 206.Thus the visible image 212 is an image of the field of view 206 from aslightly shifted viewpoint corresponding to the movement of the user'shead.

At a process 110, if the head movement has exceeded the boundary and/orthe user's head is not directed toward the current field of view, theimage on the display system changes to notify the user that the field ofview is not visible on the display system. For example, at an optionalprocess 110, the teleoperational system may enter a virtual exploremode. In another example at an optional process 112, the teleoperationalsystem may provide an alert. In the example embodiments of FIGS. 7 and 8, movement of the user's head is such that the user's eyes are nowoutside of the boundary 205 and the view direction 210 is no longerdirected toward the current field of view 206. Optionally, a message 212b may be displayed on the display system 203 to alert the user that hehas moved outside the current field of view. The message may includetext, graphics, or other images to alert the user or direct the user'shead back toward the field of view. The teleoperational system maysuspend following mode such that movement of the hand controllers do notmove the surgical instruments in the surgical environment.

Optionally, the teleoperational system may enter a virtual explore modein which image 212 a on the display system is a virtual user interfacethat may include images of an environment 214 outside the surgicalenvironment such as an external environment surrounding the patient oranother remote viewpoint, separated from the viewpoint of the imaginginstrument. Alternatively the virtual user interface may allow the useto access, for example, an interactive control menu for controllingfeatures of the teleoperational system, status information about thepatient or components of the teleoperational system, or a see-throughimage of the environment in front of his head.

At a process 114, directions may be provided to lead the user to movehis eyes back within the boundary so that the field of view 206 is againvisible. Directions may be, for example, in the form of an audiorecording providing verbal instructions for head movement, a sound thatchanges as the movement of the head changes, a set of textualinstructions, a graphical movement map, a visual homing beacon, or ascene of the field of view that changes color or focus as the directionof view moves toward the field of view. When the user's head matches thelast field of view direction, teleoperational system may exit virtualexplore mode and return to following or engagement mode. An audio,visual, or tactical indication may alert the user that virtual exploremode has been exited.

Alternatively the user may exit the virtual explore mode and re-enter anengagement mode, a following mode, or an imaging control mode at a newlycalibrated head position. The user may provide an indication that theuser is working from a new position or the system may receive anindication that a large translational movement (exceeding a thresholdmovement value and indicating that the user is working from a newposition) has been detected. Based on the indication, the system mayestablish a new position, orientation, and set of head boundaries forthe user's new location. From this new location, the position and viewdirection of the user's head may be evaluated to determine the user'scurrent field of view. For example, if a user begins by standing on oneside of a patient and then moves to the opposite side of the table, thesystem will initially enter the virtual explore mode as the user's headmoves beyond the boundary. The system may exit the virtual explore modeand re-enter a mode, such as following or imaging control mode, on theopposite side of the table. With the new head position and orientationdetected, the imaging system view displayed to the user is based on theuser's new position and orientation.

The teleoperational system may be constrained to provide a virtualimaging control mode that requires the user input prior to moving theactual or simulated imaging instrument within the surgical environment.FIG. 9 is a flowchart 300 providing a method for entering and operatingwithin a virtual imaging control mode of a teleoperational surgicalsystem. The teleoperational surgical system may be, for example ateleoperational surgical system 11 for use in a live surgicalenvironment 10 or a simulator environment 40. The method 300 isillustrated in FIG. 9 as a set of operations or processes 302 through312. Not all of the illustrated processes 302 through 312 may beperformed in all embodiments of method 300. Additionally, one or moreprocesses that are not expressly illustrated in FIG. 9 may be includedbefore, after, in between, or as part of the processes 302 through 312.In some embodiments, one or more of the processes may be implemented, atleast in part, in the form of executable code stored on non-transitory,tangible, machine-readable media that when run by one or more processors(e.g., the processors of control system) may cause the one or moreprocessors to perform one or more of the processes.

At a process 302, an image of a surgical environment visible in thefield of view of an imaging instrument is displayed. The image isdisplayed on a user-mounted immersive display system. With reference toFIG. 10 , a user 201 may want to change the current field of view 206within the potential field of view 208 defined by the range of motion ofthe imaging system 204.

At a process 304, an imaging control input is detected. The imagingcontrol input may be a clutching input that transitions theteleoperational system out of an engagement or following mode. Forexample, the image control input may be an input at a hand controller 18such as pushing a button 24. Other types of user inputs includingswitches, hand gestures, pedal movements or other type of deliberateuser input may be a suitable image control input for initiating an imagecontrol mode.

At a process 306, responsive to the detection of the imaging controlinput, the teleoperational system enters an imaging control mode. In theimaging control mode, the field of view of the imaging instrument may bechanged by moving the imaging instrument within the instrument's rangeof motion. In the imaging control mode, the surgical instruments arelocked in place and movement of the operator input system does not causea corresponding motion of the surgical instruments. In the imagingcontrol mode, the imaging instrument may be moved in response to a usermotion such as movement of the user's head or movement of operator inputsystem.

At a process 308, movement of the user's head is detected. Alternativelyanother type of user input such as movement of a hand controller 18 maybe detected. For example, with reference to FIG. 10 , a movement 211 ofthe head 201 may be detected. The view direction 210 changes with themovement of the user's head.

At a process 310, the field of view of the imaging instrument changesfrom field of view 206 to field of view 207. Field of view 207 is thefield of view of the imaging system 204 after it has been moved incorrespondence with the detected head movement 211.

At a process 312, an image of the surgical environment from the changedfield of view is displayed on the display system. For example, withreference to FIG. 10 , the image 215 a of the field of view 206 on thedisplay system 203 is changed to an image 215 b of field of view 207.

When the teleoperational system is in the imaging control mode, movementof the hand controllers 18 may cause movement of the instrument 204within the range of motion and may consequently change the field of viewof the instrument 204. Alternatively, movement of the hand controllers18 may cause ghosted images of the surgical instruments in the field ofview to move in accordance with the movement of the controllers.

If the detected motion of the user's head does not correspond to amovement within the range of motion of the imaging instrument, theteleoperational system may enter the virtual explore mode and provideimages of other environments, user interfaces, or information andpreviously described. The virtual explore mode may be exited by matchingthe user's head with the last position and orientation of the imagingsystem, as previously described.

The imaging control mode may be exited when an additional imagingcontrol input (e.g., depressing a button on the controller 18) isdetected. In various alternative embodiments, the imaging control modemay require continuous activation of the imaging control input (e.g.,continuous depression of a button on the controller 18) such thatdiscontinuing the control input transitions the teleoperational systemfrom imaging control mode into an engagement or following mode.

The teleoperational system may be constrained to provide a virtualmanipulator arm mode that indicates when a user's arm motion is outsideof a range of motion that corresponds to an allowable range of motion ofthe manipulator arm (e.g., arm 12). FIG. 11 is a flowchart 400 providinga method for entering and operating within a virtual manipulator armmode of a teleoperational system. The teleoperational surgical systemmay be, for example a teleoperational surgical system 11 for use in alive surgical environment 10 or a simulator environment 40. The method400 is illustrated in FIG. 11 as a set of operations or processes 402through 408. Not all of the illustrated processes 402 through 408 may beperformed in all embodiments of method 400. Additionally, one or moreprocesses that are not expressly illustrated in FIG. 11 may be includedbefore, after, in between, or as part of the processes 402 through 408.In some embodiments, one or more of the processes may be implemented, atleast in part, in the form of executable code stored on non-transitory,tangible, machine-readable media that when run by one or more processors(e.g., the processors of control system) may cause the one or moreprocessors to perform one or more of the processes.

At a process 402, an image of a surgical environment is displayed. Theimage is displayed on a user-mounted immersive display system. Thesurgical environment image may include an images of a surgicalinstrument and/or portions of a manipulator arm to which the instrumentis coupled.

At a process 404, a manipulator arm input motion is detected. Forexample movement of a user's arm or hand may be detected as amanipulator arm input motion. At a process 406, the teleoperationalsystem determines if the manipulator arm input motion corresponds tomovement outside of the physical manipulator arm (e.g. arm 12) range ofmotion. If the input motion corresponds to movement of the manipulatoram outside of the range of motion, the teleoperational system enters avirtual manipulator arm mode. In the virtual manipulator arm mode,movement of the user's arm does not cause corresponding motion of thephysical manipulator arm and a ghost image of the surgical instrument ora ghost image of the manipulator arm is shown moving on the displaysystem with motion corresponding to the manipulator arm input motion. Toexit the virtual manipulator arm mode, the user may be directed back tothe last allowable position of the surgical instrument or manipulatorarm. The directions back to the matched position may be similar to thoseprovided to the user to exit the virtual explore mode as previouslydescribed.

While certain exemplary embodiments of the invention have been describedand shown in the accompanying drawings, it is to be understood that suchembodiments are merely illustrative of and not restrictive on the broadinvention. Additionally, it is to be understood that the embodiments ofthe invention are not limited to the specific constructions andarrangements shown and described, since various other modifications mayoccur to those ordinarily skilled in the art.

Further, in the detailed description of the embodiments of theinvention, numerous specific details have been set forth in order toprovide a thorough understanding of the disclosed embodiments. However,it will be obvious to one skilled in the art that the embodiments ofthis disclosure may be practiced without these specific details. In someinstances, well known methods, procedures, and components have not beendescribed in detail so as not to unnecessarily obscure aspects of theembodiments of the invention.

What is claimed is:
 1. A method of operating a surgical control system,the method comprising: displaying, on a display system, an image of asurgical environment, the image including an imaging field of view, thedisplay system configured to move with a head of a user; entering animaging control mode of the surgical control system; detecting amovement of the head of the user; determining if the movement of theuser's head is within a boundary corresponding to a range of motion ofan imaging instrument, the imaging instrument having the imaging fieldof view; and while the surgical control system is in the imaging controlmode and the movement of the user's head is determined to be within theboundary, displaying a changed image of the surgical environment inresponse to the movement of the user's head, the changed image includinga changed imaging field of view, the changed imaging field of viewcorresponding to the detected movement of the user's head.
 2. The methodof claim 1 wherein entering the imaging control mode includes detectingan imaging control input from the user, the imaging control inputreceived on a hand controller communicatively coupled to the surgicalcontrol system.
 3. The method of claim 1 further comprising: disablingoperation of a surgical instrument in the surgical environment while thesurgical control system is in the imaging control mode.
 4. The method ofclaim 1 wherein entering the imaging control mode includes detecting animaging control input from the user and wherein the imaging controlinput is a continuous input.
 5. The method of claim 1 furthercomprising: if the movement of the user's head is not within theboundary corresponding to the range of motion of the imaging instrument,enabling a virtual explore mode of the surgical control system.
 6. Themethod of claim 5 further comprising: while the surgical control systemis in the virtual explore mode and responsive to the user's headmovement, displaying an image of an environment external of a patientanatomy.
 7. The method of claim 5 further comprising: while the surgicalcontrol system is in the virtual explore mode and responsive to theuser's head movement, displaying an interactive virtual user interface.8. The method of claim 5 further comprising: providing directions forexiting the virtual explore mode, wherein the directions guide a headmovement of the user to a direction that matches the imaging field ofview.
 9. The method of claim 8 further comprising: providing anindication to the user that the surgical control system has exited thevirtual explore mode.
 10. The method of claim 1 wherein the surgicalenvironment is a simulated surgical environment and the image is from asimulator imaging tool.
 11. The method of claim 1 further comprising:detecting a movement of a hand controller communicatively coupled to thesurgical control system while in the imaging control mode; and while thesurgical control system is in the imaging control mode and responsive tothe detected movement of the hand controller, displaying a secondchanged image of the surgical environment, the second changed imageincluding a second changed imaging field of view, the second changedimaging field of view corresponding to the detected movement of the handcontroller.
 12. The method of claim 1 further comprising: detecting amovement of a hand controller communicatively coupled to the surgicalcontrol system while in the imaging control mode; and while the surgicalcontrol system is in the imaging control mode and responsive to thedetected movement of the hand controller, generating a ghost image of asurgical instrument in the surgical environment, wherein the ghost imagemoves in correspondence with movement of the hand controller.
 13. Themethod of claim 1 wherein the display system is configured to mount tothe head of the user.
 14. The method of claim 1 wherein the imagingfield of view comprises a modeled field of view for a virtual endoscope.15. The method of claim 1 wherein the imaging field of view comprises afield of view of an endoscope.
 16. A system comprising: a processor; anda memory having computer readable instructions stored thereon, thecomputer readable instructions, when executed by the processor, causethe system to: display, on a display system, an image of a surgicalenvironment, the image including an imaging field of view, the displaysystem configured to move with a head of a user; enter an imagingcontrol mode of a surgical control system; detect a movement of the headof the user; determine if the movement of the user's head is within aboundary corresponding to a range of motion of an imaging instrument,the imaging instrument having the imaging field of view; and while thesurgical control system is in the imaging control mode and the movementof the user's head is determined to be within the boundary, display achanged image of the surgical environment in response to the movement ofthe user's head, the changed image including a changed imaging field ofview, the changed imaging field of view corresponding to the detectedmovement of the user's head.
 17. The system of claim 16, wherein thecomputer readable instructions, when executed by the processor, furthercause the system to: if the movement of the user's head is not withinthe boundary corresponding to the range of motion of the imaginginstrument, enable a virtual explore mode of the surgical controlsystem.
 18. The system of claim 17, wherein the computer readableinstructions, when executed by the processor, further cause the systemto: display an image of an environment external of a patient anatomywhile the surgical control system is in the virtual explore mode andresponsive to the user's head movement.
 19. The system of claim 17,wherein the computer readable instructions, when executed by theprocessor, further cause the system to: display an interactive virtualuser interface while the surgical control system is in the virtualexplore mode and responsive to the user's head movement.
 20. The systemof claim 17, wherein the computer readable instructions, when executedby the processor, further cause the system to: provide directions forexiting the virtual explore mode, wherein the directions guide a headmovement of the user to a direction that matches the imaging field ofview.